Adaptive Modeling and Planning for Reactive Agents

This research is concerned with problems where an agent is situated in a stochastic world without prior knowledge of the world’s dynamics. The agent must act in such a way so as to maximize its expected discounted reward over time. The state and action spaces are extremely large or infinite, and control decisions are made in continuous time. The objective of this research is to create a system capable of generating competent behavior in real time. The approach taken in my research is to incrementally learn a model that can be used for planning. Sutton (1990) and Moore & Atkeson (1993) used a similar approach to solve problems where the underlying world dynamics are modeled by a Markov decision process with finite states and actions and control decisions taking place at unit intervals. The parameters of the model are estimated from experience and dynamic programming (Bellman 1957) is used to produce optimal reactive plans. The problems considered in my research, however, cannot be modeled as a Markov decision process because they involve infinitely many states and actions with control decisions taking place in continuous time. One way to model such problems is to partition the state space into regions and treat all the states in the same region collectively. The action space may also be partitioned into regions. Actions in the same region are assumed to have the same effect on the world. Because transitions between regions of the state space are of variable duration, the world should be modeled by a semi-Markov decision process. The parameters of the model can be estimated from experience, and dynamic programming can be used for planning. The primary challenge in this work is determining how to partition the state and action space in a way that maximizes performance. Since it is not feasible to consider each possible way to partition these spaces, my system relies upon heuristics to determine how the partition should be adapted. This computational framework is called the Adaptive Modeling and Planning System (AMPS) since it refines its model based on experience and incorporates planning to incrementally improve behavior. This system is designed to be efficient and broadly applicable across domains.